This invention relates to a system and method for logging and tracking data at various locations and tracking movement of data. Business and legal requirements may require monitoring of network data traffic, which may include data packets flowing across the network. For example, anti-terrorism laws may require an Internet Service Provider (ISP) to maintain logs of all Internet traffic of its customers for a prescribed time period. The goals of such laws are to assist law enforcement agencies to investigate potential terrorist activities, including planning and financing. Other goals may include investigating potential lawbreakers and thwarting child pornographers and other internet predators. Investigations into illicit behavior are often hampered because such log data is routinely deleted in the normal course of business. Furthermore, the value of the current log is limited due to the fact that it contains very basic metadata (data about data) and nothing about the data traffic payload. Corporations may use this data to help them better manage their networks and to identify anomalous or unwanted network traffic. This data, however, is subject to the same limitations as described above.
Storing the entire network traffic is technically feasible, but this approach would come at great cost in terms of storage and archival. In addition, the laws of some countries may prohibit inspection of people's data without court approval or other authorization on a case by case basis. Furthermore, even if the entire traffic data were retained, there is no method to efficiently and effectively search the data. In the US, legislation has been enacted and new legislation is proposed to permit limited surveillance in the form of logging. Such logging may keep the names of an ISP's customers and their IP addresses, the IP addresses of the sites to which they connected, and the dates and times of their connections. Because the goal is investigative, the paucity of data limits the value of the log. For example, if investigators were to have the entire network traffic available for inspection, including the payload, the quality of their data would improve significantly, thus aiding their investigation. However, this is not feasible, due to various laws prohibiting such surveillance. In corporate use, the cost associated with storing all network traffic may not be justifiable.
With the proliferation of computer and communication systems there has been a significant increase in the amount of data within systems. There has also been a tremendous increase in the amount of data copied, moved, or shared between systems. As the adoption of these systems grows, the amount of data handled by and through them also increases. They become more essential to the operation, control, and management of an endless variety of use-cases including but not limited to personal, business, and governmental applications. Because of their greater use and high concentration of data-value they become targets of criminal attack and vehicles for unwanted and unauthorized activities. For example, a system may contain or handle a large amount of financial data. Due to this concentration of data, the system becomes a high-value target for criminal exploitation. Another example is that systems are being operated by criminals, vandals, terrorists, and so on, for the purpose of planning or carrying out their criminal activities such as the trade in child pornography, terrorist planning, illegal gambling, drug trafficking, and so on. Another example is that the systems of unsuspecting victims are being hijacked without authorization through remote means and used as vehicles through which further bad acts can be perpetrated.
Another example is that systems in the workplace are being used for a blend of personal and business reasons. This is a common use-case which most companies explicitly or tacitly approve of Even so, companies are being compelled by legislation to take responsibility for all communications and computer data traffic generated or made by company personnel whether or not such communications are authorized, internal, external, or personal. As an example, stock brokerages must take responsibility for the communications between their traders and their customers. Also, companies must be able to determine if any inappropriate activities are going on within their workplace. This can range from offensive language used in emails between employees to the use of company systems to store and distribute illegal copies of music, video, texts, books, and software.
In addition, there is an added challenge to logging which is the issue of privacy. Most countries have laws which respect and protect the privacy of their citizens. As such, it is typically impossible for law enforcement officials to monitor citizen data without a court order. While this restriction may help protect the privacy of the citizenry, it also blunts the benefits of such logging and investigation. Until now, the challenge has been to find sufficient evidence of wrongdoing so that a search warrant can be justified. The means for producing sufficient evidence would benefit from deep inspection into the content of data objects but this is specifically prohibited without a search warrant.
There is a need, therefore, for an improved method, article of manufacture, and apparatus for monitoring data.